H.264/MPEG-4 Part 10 or Advanced Video Coding (AVC) is the state of the art video coding standard. It is a hybrid codec which takes advantages of eliminating redundancy between frames and within one frame. The output of the encoding process is Video Coding Layer (VCL) data which is further encapsulated into Network Abstraction Layer (NAL) unit prior to transmission or storage.
Supplemental Enhancement Information (SEI) is provided to help in processes related to post-decoding, such as display. Examples of SEI messages include defining post-filter hints and tone mapping. The SEI messages are designed in such a way that they are not mandatory for the decoding process. Conforming decoders can discard SEI messages they do not understand. SEI messages are today encapsulated into a NAL unit just like VCL data, but do not affect decoding.
According to the prior art, a file must be made with specific knowledge of the media terminals display capabilities to guarantee correct playback. New display features added through enhancement information, i.e. outside the decoding loop, cannot be done in a backwards compatible manner. An example of this is the spatially interleaved frames SEI messages that have been added to H.264 video to assist in rendering decoded pictures on three-dimensional (3D) display devices. These SEI messages are derived from the effort of using an existing two-dimensional (2D) codec, such as H.264, to code 3D content. An example of this is where an SEI message describes that even frames are allocated to the left view and odd frame are allocated to the right view.
SEI messages are designed in a discardable way when it comes to decoding. It is important, however, that the media terminal can correctly interpret this information for things outside decoding, such as 3D rendering. A legacy media terminal that does not have the capability to understand these SEI messages would simply ignore them and render the decoded pictures on a 2D display resulting in significant display artifacts, e.g. hopping between left and right views every frame, for the above presented example.
Furthermore, today a media terminal needs to decode the video bit stream in order to find out whether and which types of enhancement information is necessary. It is problematic for both local playback and transmission. For example, one may have a movie that is not fully displayable and the media terminal will only find out that it cannot correctly display it when it gets to the unknown SEI message. In another case, when it comes to media transmission, a simple and quick method of thinning a bit stream is to remove all SEI messages as these are not required for decoding. This can lead to thinning of SEI messages which are important for rendering.
There is, thus, a general need within the technical field of efficiently handling supplemental processing information, such as SEI data, in order to solve or at least mitigate the problems of the prior art.